Interoperability may be the ability of a computer system or application to work with other computer systems or applications, typically without special effort on the part of the users. Although many applications exist which can expose to another application or system an external graphical user interface (GUI) for interoperability via an application programming interface (API) or a software development kit (SDK) which allow interaction with the GUI by low-level programming techniques, there exist applications which cannot be connected or have their internal events easily known by any existing technique.
An example of an application which often cannot be connected by standard API and/or SDK techniques is an application executing or running in a remote environment (e.g., in a server/client architecture). In this case, the user (e.g. using the client system) may be able to see an image of an application on the screen of the client device and perform various input operations using a keyboard and/or mouse or touchscreen, but existing application integration techniques cannot recognize user interface (UI) elements or connect to any API exposed by the application, even when such APIs exist and are available or ready to use. As such, a third party application attempting to work with or access an application may not be able to. Examples of remote environments in which this problem is prevalent include the Microsoft Remote Desktop system, the Citrix XenApp system, the PCAnywhere system, and the Oracle VM system.
Interoperability problems can be attributed to one or more of for example lack of a reliable connector (e.g. API or SDK) for an interaction with such applications; and lack of an object model exposed by GUI elements or objects (for example, buttons, list boxes, links, tables, etc.).
One prior attempt at solving the interoperability issue includes technology focused on Optical Mark Recognition (OMR), which is used with Optical Character Recognition (OCR) engines to format text while generating a specific text document. OMR also provides the ability to recognize a text document with a template designed by developers. However, OMR and similar technologies do not provide an ability to interact with an application image, among other deficiencies.
One aspect of interoperability includes working with tables or information presented in grid form (e.g., data organized in rows and columns). For example, a first application running on a client server may generate and present a table on a GUI and a second application may need to interact with the table. In many instances, however, there may not be any reliable connector (e.g. API or SDK) that would enable the second application to interact with the application that generated the table. The second application may, however, be able to obtain or receive a digital image of the table. There exist known solutions that enable recognizing tables in documents or images. A table may be recognized in the document using an OCR engine, and may be represented as formatted text or as a regular matrix with cells as its elements. However, these solutions are appropriate for static tables. Thus, these solutions may not support fluent or real time interaction with dynamic tables, or with tables that may change over time by the first application. Additionally, these solutions may not enable the second application to change the content of the table.